1. Field of the Invention
This invention relates to image processing, and more particularly, to color space conversion of patterned, color images.
2. Background
Digital cameras are commonly used today. Typically, a digital camera uses a sensor that converts light into electrical charges. The image sensor employed by some digital cameras is a charge coupled device (CCD), while others use complementary metal oxide semiconductor (CMOS) technology.
Digital cameras store pixel values for an image array as digital data. This digital data may then be transferred to a computing device, including, desktop, notebook or laptop computers, embedded systems and/or personal digital assistants for viewing. A high bandwidth connection will be required to move the image from the digital camera to a viewer at a computing system (for example, a personal computer, cell phone, or PDA) at a reasonable rate (for example, 15–30 frames per second for video), if the data is not compressed. Bandwidth may be defined in units of number of bits per pixel, number of pixels per frame, and number of frames per second. Typically, the connection speed between a digital camera and computing systems is not very high. Hence, some form of image compression, is applied prior to the transfer of data from the digital camera to computing system.
To represent a high quality color image, each pixel in a pixel array must contain tri-stimulus color data (for example, red, green, and blue). Generating such data in a digital camera requires that each light detector location in an image sensor produce data for three colors. Such a system is quite complex and expensive to build. To reduce system complexity, color image sensors commonly used in digital cameras (both video and still) use a mosaic pattern of single colored light detectors, typically red, green and blue. Each detector location produces only one of the three required colors. Such a system reduces the amount of silicon area required by the sensor, reduces the complexity of circuitry required by the system, and reduces the amount data to be managed by the system. But, these reductions come at the cost of providing insufficient data to create a full color image. Thus, additional processing is required to produce the two missing colors at each detector location (pixel).
FIG. 1 depicts one such mosaic pattern 100, commonly referred to as a “Bayer pattern”. In pattern 100, each detector location collects red (R) 103, green (G) 101, or blue (B) 102 light. Green light detectors (101) are more abundant than red (103) and blue (102) detectors to match the characteristics of human visual system. To create a full color image the two “missing colors” at each location are inferred through an interpolation process using data from neighboring detector locations. This data is further processed through a color reconstruction algorithm to make a presentable picture.
FIG. 3 shows a flow diagram of various process steps that occur in conventional image processing systems. Based on auto-focus 302B, an image sensor detects pixel values in step S300. In step S301, a dark current reduces noise introduced by the image sensor. In step S302, defect concealment is performed by an image processing unit and white balancing occurs in step S303. In step S304, color interpolation is performed. In step S305, black level flare correction is performed, and then color correction is performed in step S306. In step S307, gamma correction is performed, and edge enhancement is performed in step S308. In step S309, color space conversion occurs and then compression unit 206 (FIG. 2) compresses image data.
The most widely used compression scheme for digital cameras is provided by the JPEG standard, published by an ISO based standards organization as “International Standard ISO/IEC 15444 Part 1”, incorporated herein by reference in its entirety. The following steps are utilized by the JPEG encoding (also referred to as compression) algorithm to compress images:
Color Space Conversion: A color image in the red, green, and blue (RGB) color space contains high spatial frequency data and thus does not lend itself to high compression ratios without suffering significant information loss. A luminance/chrominance (YCbCr) space is a popular choice for a compression color space because two of the three components (the red and blue chrominance components) contain predominately low spatial frequency data and thus provide high compression ratios with minimal information loss. The third component (luminance) contains high spatial frequency data, but it is similar to the original green component image. Thus, through a simple, reversible (loss less) color space conversion (RGB to YCbCr) larger compression ratios are achieved. One such set of equations (expressed in matrix form) to convert from RGB to YCbCr color space are:
                              [                                          ⁢                                                    Y                                                                                      C                  ⁢                                                                          ⁢                  b                                                                                                      C                  ⁢                                                                          ⁢                  r                                                              ⁢                                          ]                =                                                            1                256                            [                                                          ⁢                                                                    65.738                                                        129.057                                                        25.064                                                                                                              -                      37.945                                                                                                  -                      74.494                                                                            112.439                                                                                        112.439                                                                              -                      94.154                                                                                                  -                      18.285                                                                                  ⁢                                                          ]                        ×                          [                                                          ⁢                                                                    R                                                                                        G                                                                                        B                                                              ⁢                                                          ]                                +                      [                                                  ⁢                                                            16                                                                              128                                                                              128                                                      ⁢                                                  ]                                              (        I        )            
Various other sets of equations may also be used such as those specified by the ITU Radiocommunication Sector (formerly CCIR). (A Technical Introduction to Digital Video by Charles A. Poynton, John Wiley & Sons, Inc. publisher, 1996, TSBN:0-471-12253-X)
Subsampling: Since red and blue chrominance components contain predominately low frequency data they are sub-sampled in either vertical or horizontal dimensions, or in both dimensions to achieve a significant amount of compression without undue image degradation.
Frequency Transformation: Images captured by digital cameras contain a large amount of redundant spatial information when viewed by an observer. This redundant information can be removed without detection by the human observer thus increasing the compression ratio. To identify this redundant information, image data is transformed from its native spatial representation to a signal frequency based representation via a Discrete Cosine Transform. This is transformation reversible (loss less).
Quantization: With the image represented in frequency space, redundant information is identified and removed by quantizing (in frequency space) pixel values to predetermined levels. These levels (quantization coefficients) determine the amount of compression and the amount of image quality degradation. Higher levels of quantization result in larger compression ratios (smaller encoded images) and more degradation. The quantization process is irreversible (lossy).
Entropy Coding: The quantization step results in large numbers of pixel values (in frequency space) being reduced to small values. Using variable code length techniques (e.g. Huffman encoding) these small values are encoded based on their predicted frequency of occurrence. Values that occur more often are assigned short bit sequences, and those that occur less often are assigned long bit sequences. The entropy coding process is reversible (loss less).
Compressed images must be decompressed before being viewed. The following steps are utilized by JPEG decoding (decompression) algorithm for decompressing a compressed image:
Entropy Decoding: Using techniques from the field information theory (e.g. Huffman decoding) variable length bit encoded bit strings are converted to quantized frequency coefficients.
Dequantization: Quantized frequency coefficients are expanded (dequantized) to values within the range of the originally derived (prior to quantization during encoding) values. Since the quantization step is lossy, the exact original values are, in general, not recovered. Instead, close approximations (based on the amount of compression loss) are recovered.
Spatial Transformation: Image displays require data to be represented in the spatial domain, not the frequency domain. This conversion back to the spatial domain from the frequency domain is achieved via the Inverse Discrete Cosine Transform.
Up-sampling: If the red and blue chrominance component images were sub-sampled during encoding, they are up-sampled by duplicating each pixel value in either the horizontal or vertical dimension, or in both dimensions.
Color Space Conversion: Most digital image displays (for example, computer monitors) require image data to be in the RGB color space. The equations to convert from YCbCr (luminance/chrominance) color space to RGB color space can be expressed in matrix form as follows:
                              [                                          ⁢                                                    R                                                                    G                                                                    B                                              ⁢                                          ]                =                                                            1                256                            [                                                          ⁢                                                                    298.082                                                        0.0                                                        408.583                                                                                        298.082                                                                              -                      100.291                                                                                                  -                      208.120                                                                                                            298.082                                                        516.411                                                        0.0                                                              ⁢                                                          ]                        ×                          [                                                          ⁢                                                                    Y                                                                                                              C                      ⁢                                                                                          ⁢                      b                                                                                                                                  C                      ⁢                                                                                          ⁢                      r                                                                                  ⁢                                                          ]                                -                      [                                                  ⁢                                                            16                                                                              128                                                                              128                                                      ⁢                                                  ]                                              (        II        )            
Various other sets of equations may also be used such as those specified by the ITU Radiocommunication Sector (formerly CCIR). (See A Technical Introduction to Digital Video by Charles A. Poynton, John Wiley & Sons, Inc. publisher, 1996, ISBN:0-471-12253-X). It is noteworthy that RGB to YCbCr and YCbCr to RGB equations are a “matched set” to ensure that no additional information loss occurs.
Although the JPEG compression technique has been described above, other compression schemes for example, MPEG, Wavelet, Loss less JPEG may be also be used to achieve high compression rates with minimal image quality degradation. These techniques typically use similar steps as those described above, especially the Color Space Conversion. The specific coefficients used in the color space conversion however, may vary from those given above.
Conventional image processing techniques used in digital cameras have disadvantages. For example, as shown by FIG. 3 process steps, various image processing steps are performed before data is compressed after step S309. These steps use resources and slow down the process.
Also, once a full color image is available after color interpolation and reconstruction processing is complete, the amount of data is increased three times after the reconstruction. In conventional techniques, for example, in a 1 Mega pixel digital camera, sensor data is first converted to 3 million bytes (although raw image data is for 1 million pixels for one color only, the other two colors are estimated at each location) and then each pixel is converted from RGB to YCbCr and the data is compressed. Not only the data is tripled from 1 million bytes to 3 million bytes, the compressed data also results in poor edge quality.
Therefore, there is a need for a method that efficiently converts mosaic image data into a color space that can efficiently compress the mosaic image data.